Vienna Talking Faces (ViTaFa): A multimodal person database with synchronized videos, images, and voices.

Social perception relies on different sensory channels, including vision and audition, which are specifically important for judgements of appearance. Therefore, to understand multimodal integration in person perception, it is important to study both face and voice in a synchronized form. We introduce the Vienna Talking Faces (ViTaFa) database, a high-quality audiovisual database focused on multimodal research of social perception. ViTaFa includes different stimulus modalities: audiovisual dynamic, visual dynamic, visual static, and auditory dynamic. Stimuli were recorded and edited under highly standardized conditions and were collected from 40 real individuals, and the sample matches typical student samples in psychological research (young individuals aged 18 to 45). Stimuli include sequences of various types of spoken content from each person, including German sentences, words, reading passages, vowels, and language-unrelated pseudo-words. Recordings were made with different emotional expressions (neutral, happy, angry, sad, and flirtatious). ViTaFa is freely accessible for academic non-profit research after signing a confidentiality agreement form via https://osf.io/9jtzx/ and stands out from other databases due to its multimodal format, high quality, and comprehensive quantification of stimulus features and human judgements related to attractiveness. Additionally, over 200 human raters validated emotion expression of the stimuli. In summary, ViTaFa provides a valuable resource for investigating audiovisual signals of social perception.

Alteration of the temporal association between courtship audio and visual components affects female sexual response.

Some multimodal signals-that is, occurring in more than one sensory modality-appear to carry additional information which is not present when component signals are presented separately. To understand the function of male ring dove's (Streptopelia risoria) multimodal courtship, we used audiovisual playback of male displays to investigate female response to stimuli differing in their audiovisual timing. From natural courtship recordings, we created a shifted stimulus where audio was shifted relative to video by a fixed value and a jittered stimulus, where each call was moved randomly along the visual channel. We presented 3 groups of females with the same stimulus type, that is, control, shifted, and jittered, for 7 days. We recorded their behavior and assessed pre- and post-test blood estradiol concentration. We found that playback exposure increased estradiol levels, confirming that this technique can be efficiently used to study doves' sexual communication. Additionally, chasing behavior (indicating sexual stimulation) increased over experimental days only in the control condition, suggesting a role of multimodal timing on female response. This stresses the importance of signal configuration in multimodal communication, as additional information is likely to be contained in the temporal association between modalities.

The Effects of Pitch Manipulation on Male Ratings of Female Speakers and Their Voices.

Vocal and facial cues typically co-occur in natural settings, and multisensory processing of voice and face relies on their synchronous presentation. Psychological research has examined various facial and vocal cues to attractiveness as well as to judgements of sexual dimorphism, health, and age. However, few studies have investigated the interaction of vocal and facial cues in attractiveness judgments under naturalistic conditions using dynamic, ecologically valid stimuli. Here, we used short videos or audio tracks of females speaking full sentences and used a manipulation of voice pitch to investigate cross-modal interactions of voice pitch on facial attractiveness and related ratings. Male participants had to rate attractiveness, femininity, age, and health of synchronized audio-video recordings or voices only, with either original or modified voice pitch. We expected audio stimuli with increased voice pitch to be rated as more attractive, more feminine, healthier, and younger. If auditory judgements cross-modally influence judgements of facial attributes, we additionally expected the voice pitch manipulation to affect ratings of audiovisual stimulus material. We tested 106 male participants in a within-subject design in two sessions. Analyses revealed that voice recordings with increased voice pitch were perceived to be more feminine and younger, but not more attractive or healthier. When coupled with video recordings, increased pitch lowered perceived age of faces, but did not significantly influence perceived attractiveness, femininity, or health. Our results suggest that our manipulation of voice pitch has a measurable impact on judgements of femininity and age, but does not measurably influence vocal and facial attractiveness in naturalistic conditions.

Art and Perception: Using Empirical Aesthetics in Research on Consciousness.

Understanding consciousness is a major frontier in the natural sciences. However, given the nuanced and ambiguous sets of conditions regarding how and when consciousness appears to manifest, it is also one of the most elusive topics for investigation. In this context, we argue that research in empirical aesthetics-specifically on the experience of art-holds strong potential for this research area. We suggest that empirical aesthetics of art provides a more exhaustive description of conscious perception than standard laboratory studies or investigations of the less artificial, more ecological perceptual conditions that dominate this research, leading to novel and better suited designs for natural science research on consciousness. Specifically, we discuss whether empirical aesthetics of art could be used for a more adequate picture of an observer's attributions in the context of conscious perception. We point out that attributions in the course of conscious perception to (distal) objects versus to media (proximal objects) as origins of the contents of consciousness are typically swift and automatic. However, unconventional or novel object-media relations used in art can bring these attributions to the foreground of the observer's conscious reflection. This is the reason that art may be ideally suited to study human attributions in conscious perception compared to protocols dedicated only to the most common and conventional perceptual abilities observed under standard laboratory or "natural"/ecological conditions alone. We also conclude that art provides an enormous stock of such unconventional and novel object-media relations, allowing more systematic falsification of tentative conclusions about conscious perception versus research protocols covering more conventional (ecological) perception only. We end with an outline of how this research could be carried out in general.

Forgotten rhythms? Revisiting the first evidence for rhythms in cognition.

Practically every neuroscientist knows that human brain rhythms were first recorded in the 1920s by Hans Berger, who coined the term 'alpha waves' for the regular activity of around 10 cycles per second that was clearly visible in many of his recordings. Almost 100 years later, alpha rhythms are still the subject of active investigation and continue to intrigue researchers. What we have perhaps forgotten though, is the clever experimentation that was carried out during the first decades of electroencephalogram (EEG) research, often using sophisticated, custom-made analysis and stimulation devices. Here, I review selected findings from the early EEG literature regarding the character, origin, and meaning of human brain rhythms, beginning with Berger's publications and then focusing on the use of regular visual stimulation as a tool to understand intrinsic brain rhythms. It is clear that many of these findings are still relevant to open questions about the role of rhythmic brain activity. In addition, they also contain some general lessons for contemporary neuroscientists, meaning that there is great value in looking back at these forgotten publications.

Video Recording and Analysis of Avian Movements and Behavior: Insights from Courtship Case Studies.

Video recordings are useful tools for advancing our understanding of animal movements and behavior. Over the past decades, a burgeoning area of behavioral research has put forward innovative methods to investigate animal movement using video analysis, which includes motion capture and machine learning algorithms. These tools are particularly valuable for the study of elaborate and complex motor behaviors, but can be challenging to use. We focus in particular on elaborate courtship displays, which commonly involve rapid and/or subtle motor patterns. Here, we review currently available tools and provide hands-on guidelines for implementing these techniques in the study of avian model species. First, we suggest a set of possible strategies and solutions for video acquisition based on different model systems, environmental conditions, and time or financial budget. We then outline the available options for video analysis and illustrate how different analytical tools can be chosen to draw inference about animal motor performance. Finally, a detailed case study describes how these guidelines have been implemented to study courtship behavior in golden-collared manakins (Manacus vitellinus).

Cholinergic manipulations affect sensory responses but not attentional enhancement in macaque MT.

BACKGROUND: Attentional modulation in the visual cortex of primates is characterized by multiplicative changes of sensory responses with changes in the attentional state of the animal. The cholinergic system has been linked to such gain changes in V1. Here, we aim to determine if a similar link exists in macaque area MT. While rhesus monkeys performed a top-down spatial attention task, we locally injected a cholinergic agonist or antagonist and recorded single-cell activity. RESULTS: Although we confirmed cholinergic influences on sensory responses, there was no additional cholinergic effect on the attentional gain changes. Neither a muscarinic blockage nor a local increase in acetylcholine led to a significant change in the magnitude of spatial attention effects on firing rates. CONCLUSIONS: This suggests that the cellular mechanisms of attentional modulation in the extrastriate cortex cannot be directly inferred from those in the primary visual cortex.

Song Is More Memorable Than Speech Prosody: Discrete Pitches Aid Auditory Working Memory.

Vocal music and spoken language both have important roles in human communication, but it is unclear why these two different modes of vocal communication exist. Although similar, speech and song differ in certain design features. One interesting difference is in the pitch intonation contour, which consists of discrete tones in song, vs. gliding intonation contours in speech. Here, we investigated whether vocal phrases consisting of discrete pitches (song-like) or gliding pitches (speech-like) are remembered better, conducting three studies implementing auditory same-different tasks at three levels of difficulty. We tested two hypotheses: that discrete pitch contours aid auditory memory, independent of musical experience ("song memory advantage hypothesis"), or that the higher everyday experience perceiving and producing speech make speech intonation easier to remember ("experience advantage hypothesis"). We used closely matched stimuli, controlling for rhythm and timbre, and we included a stimulus intermediate between song-like and speech-like pitch contours (with partially gliding and partially discrete pitches). We also assessed participants' musicality to evaluate experience-dependent effects. We found that song-like vocal phrases are remembered better than speech-like vocal phrases, and that intermediate vocal phrases evoked a similar advantage to song-like vocal phrases. Participants with more musical experience were better in remembering all three types of vocal phrases. The precise roles of absolute and relative pitch perception and the influence of top-down vs. bottom-up processing should be clarified in future studies. However, our results suggest that one potential reason for the emergence of discrete pitch-a feature that characterises music across cultures-might be that it enhances auditory memory.

Evolution and function of multimodal courtship displays.

Courtship displays are behaviours aimed to facilitate attraction and mating with the opposite sex and are very common across the animal kingdom. Most courtship displays are multimodal, meaning that they are composed of concomitant signals occurring in different sensory modalities. Although courtship often strongly influences reproductive success, the question of why and how males use multimodal courtship to increase their fitness has not yet received much attention. Very little is known about the role of different components of male courtship and their relative importance for females. Indeed, most of the work on courtship displays have focused on effects on female choice, often neglecting other possible roles. Additionally, a number of scientists have recently stressed the importance of considering the complexity of a display and the interactions between its different components in order to grasp all the information contained in those multimodal signals. Unfortunately, these methods have not yet been extensively adapted in courtship studies. The aim of this study was to review what is currently known about the functional significance of courtship displays, particularly about the role of multimodality in the courtship communication context. Emphasis is placed on those cases where a complete picture of the communication system can only be assessed by taking complexity and interaction between different modalities into account.

A Pressure Injection System for Investigating the Neuropharmacology of Information Processing in Awake Behaving Macaque Monkey Cortex.

The top-down modulation of feed-forward cortical information processing is functionally important for many cognitive processes, including the modulation of sensory information processing by attention. However, little is known about which neurotransmitter systems are involved in such modulations. A practical way to address this question is to combine single-cell recording with local and temporary neuropharmacological manipulation in a suitable animal model. Here we demonstrate a technique combining acute single-cell recordings with the injection of neuropharmacological agents in the direct vicinity of the recording electrode. The video shows the preparation of the pressure injection/recording system, including preparation of the substance to be injected. We show a rhesus monkey performing a visual attention task and the procedure of single-unit recording with block-wise pharmacological manipulations.

Feature-selective attention in healthy old age: a selective decline in selective attention?

Deficient selection against irrelevant information has been proposed to underlie age-related cognitive decline. We recently reported evidence for maintained early sensory selection when older and younger adults used spatial selective attention to perform a challenging task. Here we explored age-related differences when spatial selection is not possible and feature-selective attention must be deployed. We additionally compared the integrity of feedforward processing by exploiting the well established phenomenon of suppression of visual cortical responses attributable to interstimulus competition. Electroencephalogram was measured while older and younger human adults responded to brief occurrences of coherent motion in an attended stimulus composed of randomly moving, orientation-defined, flickering bars. Attention was directed to horizontal or vertical bars by a pretrial cue, after which two orthogonally oriented, overlapping stimuli or a single stimulus were presented. Horizontal and vertical bars flickered at different frequencies and thereby elicited separable steady-state visual-evoked potentials, which were used to examine the effect of feature-based selection and the competitive influence of a second stimulus on ongoing visual processing. Age differences were found in feature-selective attentional modulation of visual responses: older adults did not show consistent modulation of magnitude or phase. In contrast, the suppressive effect of a second stimulus was robust and comparable in magnitude across age groups, suggesting that bottom-up processing of the current stimuli is essentially unchanged in healthy old age. Thus, it seems that visual processing per se is unchanged, but top-down attentional control is compromised in older adults when space cannot be used to guide selection.

Competitive interactions of attentional resources in early visual cortex during sustained visuospatial attention within or between visual hemifields: evidence for the different-hemifield advantage.

Performing a task across the left and right visual hemifields results in better performance than in a within-hemifield version of the task, termed the different-hemifield advantage. Although recent studies used transient stimuli that were presented with long ISIs, here we used a continuous objective electrophysiological (EEG) measure of competitive interactions for attentional processing resources in early visual cortex, the steady-state visual evoked potential (SSVEP). We frequency-tagged locations in each visual quadrant and at central fixation by flickering light-emitting diodes (LEDs) at different frequencies to elicit distinguishable SSVEPs. Stimuli were presented for several seconds, and participants were cued to attend to two LEDs either in one (Within) or distributed across left and right visual hemifields (Across). In addition, we introduced two reference measures: one for suppressive interactions between the peripheral LEDs by using a task at fixation where attention was withdrawn from the periphery and another estimating the upper bound of SSVEP amplitude by cueing participants to attend to only one of the peripheral LEDs. We found significantly greater SSVEP amplitude modulations in Across compared with Within hemifield conditions. No differences were found between SSVEP amplitudes elicited by the peripheral LEDs when participants attended to the centrally located LEDs compared with when peripheral LEDs had to be ignored in Across and Within trials. Attending to only one LED elicited the same SSVEP amplitude as Across conditions. Although behavioral data displayed a more complex pattern, SSVEP amplitudes were well in line with the predictions of the different-hemifield advantage account during sustained visuospatial attention.

Independent effects of attentional gain control and competitive interactions on visual stimulus processing.

Attention filters behaviorally relevant stimuli from the constant stream of sensory information comprising our environment. Research into underlying neural mechanisms in humans suggests that visual attention biases mutual suppression between stimuli resulting from competition for limited processing resources. As a consequence, processing of an attended stimulus is facilitated. This account makes 2 assumptions: 1) An attended stimulus is released from mutual suppression with competing stimuli and 2) an attended stimulus experiences greater gain in the presence of competing stimuli than when it is presented alone. Here, we tested these assumptions by recording frequency-tagged potentials elicited in early visual cortex that index stimulus-specific processing. We contrasted the processing of a given stimulus when its location was attended or unattended and in the presence or the absence of a nearby competing stimulus. At variance with previous findings, competition similarly suppressed processing of attended and unattended stimuli. Moreover, the magnitude of attentional gain was comparable in the presence or the absence of competing stimuli. We conclude that visuospatial selective attention does not directly modulate mutual suppression between stimuli but instead acts as a signal gain, which biases processing toward attended stimuli independent of competition.

Keeping focused: sustained spatial selective visual attention is maintained in healthy old age.

A better understanding of age-related change in the attentional modulation of perceptual processing may help elucidate cognitive change. For example, increased cognitive interference due to inappropriate processing of irrelevant information has been suggested to contribute to cognitive decline. However, it is not yet clear whether interference effects observed at later stages, such as executive function or response selection, are caused by leaky attentional selection at early, sensory stages of processing. Here, we investigated attentional control of sensory selection by comparing younger and older adults' ability to sustain spatial selective attention to one of two centrally presented, overlapping rapid serial visual presentation (RSVP) letter sequences, one large and one small. These stimuli elicited separable steady-state visual evoked potentials (SSVEP), which provide an index of early visual processing for each stimulus separately and are known to be modulated by selective attention. The condition of most interest required participants to attend to the larger letters while ignoring the smaller letters, as these foveally presented irrelevant stimuli were expected to present the strongest interference. Although the rapid presentation rates made the task demanding, detection ability did not differ between young and older adults. Accordingly, attentional modulation of SSVEP amplitudes was found in both age groups. Neither the magnitude nor the cortical sources of these SSVEP attention effects differed between age groups. Our results thus suggest that in the current task, the effect of voluntary spatial attention on sustained sensory processing in early visual areas is maintained in healthy old age.

Effects of overt and covert attention on the steady-state visual evoked potential.

Flickering stimuli evoke an oscillatory brain response with the same frequency as the driving stimulus, the so-called steady-state visual evoked potential (SSVEP). SSVEPs are robust brain signals whose amplitudes are enhanced with attention and thus play a major role in the development and use of non-invasive Brain-Computer Interfaces (BCIs). We compared the modulation of SSVEP amplitudes when subjects directly gazed at a flickering array of static dots (overt attention) to when they covertly shifted attention to the dots keeping their eyes at central fixation. A discrimination task was performed at the attended location to ensure that subjects shifted attention as instructed. Horizontal eye movements (allowed in overt attention but to be avoided in covert attention) were monitored by the horizontal electrooculogram. Subjects' behavioural performance was significantly reduced in covert attention compared to overt attention. Correspondingly, attentional modulation of SSVEP amplitudes by overt attention was larger in magnitude than for covert attention. Overt attention also changed the topographical distribution of SSVEP amplitudes on the scalp. Stimuli elicited the largest amplitudes at central occipital electrodes when they were overtly attended and at contralateral parieto-occipital sites when they were covertly attended. Accordingly, source analysis revealed clear centrally located sources in early visual areas in overt attention, regardless of the attended visual hemifield. Taken together these results affirm that overt and covert attention have qualitatively and quantitatively different effects on SSVEP responses as well as on task performance. Moreover, our results suggest that navigating SSVEP-BCIs with overt attention is more reliable and highlight some of the challenges in developing BCIs for patients who have lost the ability to move their eyes.

Feature-selective attention: evidence for a decline in old age.

Although attention in older adults is an active research area, feature-selective aspects have not yet been explicitly studied. Here we report the results of an exploratory study involving directed changes in feature-selective attention. The stimuli used were two random dot kinematograms (RDKs) of different colours, superimposed and centrally presented. A colour cue with random onset after the beginning of each trial instructed young and older subjects to attend to one of the RDKs and detect short intervals of coherent motion while ignoring analogous motion events in the non-cued RDK. Behavioural data show that older adults could detect motion, but discriminated target from distracter motion less reliably than young adults. The method of frequency tagging allowed us to separate the EEG responses to the attended and ignored stimuli and directly compare steady-state visual evoked potential (SSVEP) amplitudes elicited by each stimulus before and after cue onset. We found that younger adults show a clear attentional enhancement of SSVEP amplitude in the post-cue interval, while older adults' SSVEP responses to attended and ignored stimuli do not differ. Thus, in situations where attentional selection cannot be spatially resolved, older adults show a deficit in selection that is not shared by young adults.

Visual stimulus locking of EEG is modulated by temporal congruency of auditory stimuli.

Disparate sensory streams originating from a common underlying event share similar dynamics, and this plays an important part in multisensory integration. Here we investigate audiovisual binding by presenting continuously changing, temporally congruent and incongruent stimuli. Recorded EEG signals are used to quantify spectrotemporal and waveform locking of neural activity to stimulus dynamics. Spectrotemporal analysis reveals locking to visual stimulus dynamics in both a broad alpha and the beta band. The properties of these effects suggest they are a correlate of bottom-up processing in the visual system. Waveform locking reveals two cortically distinct processes that lock to visual stimulus dynamics with differing topographies and time lags relative to the stimuli. Most importantly, these are modulated in strength by the congruency of an accompanying auditory stream. In addition, the waveform locking found at occipital electrodes shows an increase over stimulus duration for visual and congruent audiovisual stimuli. Hence we argue that these effects reflect audiovisual interaction. We thus propose that spectrotemporal and waveform locking reflect different mechanisms involved in the processing of dynamic audiovisual stimuli.